Magic Music Desk: A Tangible and Ubiquitous Multimodal Music and Speech System

This paper describes a novel multi-modal multi-user audio-visual interface – the Magic Music Desk (MMD) which employs the principles of embodied interaction, and emphasizes social interaction between users. This research is aimed for use in wearable and mobile sound and music systems. We combine multiple modalities for the interfaces including speech recognition, hand gesture recognition, sound and visual mixed reality technologies. A new type of interaction which is called as “What You Say is What You See” (WYSIWYS) is demonstrated in our system. This interaction enables all users to visualize each others spoken words as 3D objects which could be seen by multi-users (this also allows multi-cultural social interaction).